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ABSTRACT 



This paper describes the creation and evaluation of a World 
Wide Web-based courseware module, using conceptual models based on 
constructivism, that teaches novices how to use the Internet for searching. 
Questionnaires and interviews were used to understand the difficulties of a 
group of novices. The conceptual model of the experts for the task was 
derived using Applied Cognitive Task Analysis (ACTA) and analogy developed to 
assist novices in understanding the expert cognitive model . The Web 
courseware module was evaluated using a second novice population. The outcome 
demonstrated that using conceptual models in learning could help novices to 
acquire an understanding of the search mechanism and so enable them to learn 
better. (Contains 11 references.) (MES) 
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found or most of the retrieved information is not relevant. To help overcome this we have 
developed a web-based courseware module, using conceptual models based on constructivism, to 
teach novices how to use the Internet for searching. This paper describes the creation and 
evaluation of the courseware. Questionnaires and interviews were used to understand the 
difficulties of a group of novices. The conceptual model of the experts for the task was derived 
using Applied Cognitive Task Analysis (ACTA) and analogy developed to assist novices 
understand the expert cognitive model. The web-courseware module was evaluated using a second 
novice population. The outcome demonstrated that using conceptual models in learning can help 
novices to acquire an understanding of the search mechanism and so enable them to learn better. 



Introduction 



To be able to use the Internet effectively is not a trivial task. The problem is further compounded when we have to 
teach others to search for information on the Internet. Part of the difficulty of teaching others to use the Internet is 
deciding what to teach and how best to teach them. There are several overlapping knowledge domains in which 
users must have some conceptual knowledge to be able to successfully use the Internet to search for information. 
Users must possess information retrieval skills, knowledge of how the system functions, knowledge of the subject in 
which they are seeking information and problem-solving skills (Brandt 1997). These problems are not helped by the 
fact that browsing the Internet can seem different from using the ’normal' office applications on the PC. Even people 
who are proficient in basic computing skills and have lengthy experience of using traditional software can be 
confronted by many problems when using the Internet. It appears that the perception that Internet is very different 
hinders the transfer of appropriate knowledge from the off-line to the on-line domain. What is needed is conceptual 
knowledge of the Internet and the tasks that can be performed on it. How, then, can we help novices to search for 
relevant information effectively on the Internet? 

We believe that conceptual models can be used to help novices to learn the Internet effectively in a constructivist 
learning environment. This paper describes the development of conceptual models using the Applied Cognitive 
Task Analysis (ACTA) method to teach Internet search. The conceptual models developed were integrated into a 
web-based courseware module designed using the Courseware Engineering Methodology (Uden 1997, 1998). 
Evaluation of the conceptual models was carried out by novices. 



Constructivist Learning 



Constructivist learning holds that learners learn new knowledge by refining to things already known or experienced. 
Constructivist teaching seeks to help learners to construct advanced knowledge that will support problem-solving 
skills and expert task performance. It is ideal for teaching a complex and ill-structured domain such as the Internet. 
According to constructivists, users construct knowledge about a new system by transferring and expanding existing 
knowledge through mental models (Brandt 1997). A mental model of a typewriter has been used to teach word 
processing. With a conceptual understanding of typing, novices could get past the confusion of an abstract electronic 
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tool and relate editing to manual typing. When learners are provided with conceptual models and situated 
experiences that reinforce those models, their mental models will be extended and developed. 

Mental Models and Conceptual Models 

A mental model is developed as a user interacts with a topic. Users create new mental models to describe the topic 
by means of structural mapping; that is, users map the structural relations of an existing model onto the new 
(Gentner & Gentner 1983). The conceptual model of the topic is created by the teacher and encompasses the 
knowledge that is to be transferred to the learners. The conceptual model is usually represented by an analogy or 
metaphor. Analogy is extremely powerful because it presents a description of a topic by reference to existing 
knowledge held by the user. Furthermore, interaction with a topic forces the user to compare the similarities and 
dissimilarities between it and the analogy presented. We believe that the development of good conceptual models 
can help to bridge the gap that exists between novice and expert Internet users. Thus, the goal of the teacher or 
designer is to seek a conceptual model of the topic that matches up with the user mental model to facilitate learning. 

There is evidence provided by researchers that conceptual models are useful in the learning of complex tasks. Mayer 
(1989) concluded that providing concrete conceptual models for learners improves conceptual retention, reduces 
verbatim recall and improves problem-solving transfer. Good conceptual models make intuitive sense to the learners 
and use vocabulary and concepts that are appropriate for the learners. The reason for intentionally illustrating the 
conceptual components to enhance learners' mental models of the content being structured. Conceptual models can 
be effectively presented before instruction or during instruction. Another reason for providing learners with the 
conceptual models of the problem solving domain being studied is because they explicitly represent the structural 
knowledge required to support problem solving (Jonassen et al. 1993). 



Capturing Novices* mental Models 



The first task of our study was to assess the mental models of the novices through familiarity with their 
backgrounds. As it is not possible to obtain an accurate mental model for all users, we would only consider where 
mental models are incomplete or erroneous, by researching the classes of problems that novices have when using the 
Internet technologies. We conducted surveys with an initial set of subjects by administering a questionnaire. This 
had 3 main sections of mainly multiple choice questions addressing the subject's background, computer experience 
and Internet experience. 63 complete questionnaires were returned by 47 high school students, 4 teachers, 2 
university students, and 10 members of the general public. Of these 3 high school students were clearly identified as 
experts and these were omitted from further study. Of the remaining subjects 53 had up to 1 years' experience of the 
Internet and 3 years’ experience of personal computers. The questionnaire provided an opportunity to discover 
novices' common problems and perceptions of the Internet. This allowed us to identify the most appropriate 
questions to ask during the follow-up interviews. The follow-up interview questions focused on gathering 
information about subjects' conceptual knowledge to help us identify current levels of understanding and 
deficiencies in the mental models of the subjects. This information was required to aid design of conceptual models 
of the topic that will connect with the subjects’ mental models. 

The follow-up interview questions concerned the problems that subjects encountered when using computers and the 
Internet. In particular, questions concerned problems of navigation (both on-line and off-line), such as getting lost 
through folders (off-line) and getting lost on the Internet. We also sought to discover how many subjects used 
keyboard shortcuts or the right mouse button to access menus. These are important indicators of lack of computing 
ability and mental model development. Only three subjects mentioned shortcuts. It would appear that novices did 
not realize that it is possible to use shortcuts when browsing the Internet. Other questions addressed the task of 
moving data (either text or image) from a web page to another document. Only two subjects mentioned copy and 
paste as a method of doing this, indicating general lack of conceptual knowledge about cutting and pasting. This tied 
in with information gained in the follow-up interviews that subjects did not perceive using the Internet to be similar 
to using the computer in other familiar situations. It would appear that mental models of computer usage were not 
being employed and developed when browsing the Internet. 
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Searching on the Internet caused a large number of problems for the subjects. They had little or no concept of 
searching as a process and the effect of syntax, semantics and Boolean logic on the results returned. Our follow-up 
interviews confirmed a gap in the subjects' conceptual knowledge of the search mechanism. Many did not see the 
link between the Internet search activity and other search methods used. In this respect it was interesting that no-one 
suggested the Tile finder' application on their personal computer was similar to Internet-based searching, which 
implied that the subjects were unable to link their mental models of off-line searching to on-line tasks. 

The subjects had little or no concept of how a web page was down-loaded onto their computers. Mental models are 
clearly under-developed in this area, which was identified as an area for the development of a conceptual model. 
With a model of the web page downloading process, subjects would be able to understand issues that affect the 
speed of transmission of data or occurrences such as the HTTP 404 'File not found' error. 

Difficulties were identified with the Internet address, the Uniform Resource Locator (URL). The subjects had little 
or no idea what the constituent parts of the URL mean, which can hamper problem solving. Specifically, they did 
not recognize the parallels between a house address made up of a number of elements that provide different levels of 
information about the location (street, city, country) of the house, and a URL having distinct elements that identify a 
particular computer, the domain that computer resides in and the service that is required of it. Furthermore, the 
subjects were not aware that, in effect, the Internet is one big (and very dynamic) file system and that the slash 
separators in an address signify directories on the remote computer. Consequently, they could not apply their mental 
models of file systems and file manipulations. This impacts upon their problem-solving abilities. For example, when 
novices come across broken links, or cannot find a file or page they expect to find, they often do not know how to go 
about solving the problem. By applying knowledge of file systems and directory structures to the structure of URLs, 
users can strip parts of the address in the hope of being presented with a directory structure to browse. 



Development of Conceptual Models 



The conceptual model of the topic is the teacher's representation of the knowledge to be transferred. In order for a 
teacher's conceptual model to be effective in facilitating learning, Norman (1983) proposed that the conceptual 
model should meet three basic criteria: leamability, functionality and usability. Leamability means that the 
conceptual model should be easy to leam. Functionality means that the model corresponds to important aspects of 
the topic. Usability implies that the model should be easy to use, given the limitations of the human information 
processing system. 

In designing conceptual models, designers must take into account the kind of things an expert considers, such as 
what problems and errors are likely to be encountered in a given situation. This conceptual model thus gives context 
and meaning to the topic, which further allows facilitation of knowledge through connection with novices mental 
models. In order to develop conceptual models to help novices in their understanding of the Internet search, we have 
to interview experts. This is necessary because a conceptual model must provide an accurate and consistent 
representation of the subject to be learned. 



Applied Cognitive Task Analysis (ACTA) 



As the activities involved in searching the Internet are complex cognitive tasks, we chose the ACTA method to elicit 
the cognitive skills or mental demands needed to perform the task proficiency from the experts. The main reason for 
choosing ACTA is that it is a streamlined version of analyzing complex cognitive tasks. It is designed to be more 
useful for the design of actual systems and easier to perform than other cognitive task analysis methods. ACTA is a 
structured interview method for eliciting appropriate task knowledge from subject matter experts (Militello & 
Hutton 1998). As a methodology, ACTA is particularly suited to the generation of learning objectives and the 
revision of existing, or creation of new, training manuals. The three interview types that comprise ACTA are the 
task diagram, the knowledge audit and the simulation interview. The interviews aim to help subject-matter experts 
verbalize the cognitive strategies used in task performance. 



Step 1. The Task Diagram: The task diagram is the preliminary interview and is intended to elicit a broad overview 
of the task or tasks. The first step of ACTA begins by asking the subject matter experts to break down a particular 
task of interest into four or five steps and to highlight which of the steps requires different cognitive skills. For our 
study two subject-matter experts, both university lecturers, were interviewed, the information collected was 
documented and a task diagram constructed. 

Step 2. The Knowledge Audit: The knowledge audit step elicits the aspects of expertise required for particular tasks 
or subtasks. The interview is based upon 'knowledge categories* such as diagnosing, predicting, perceptual skills, 
improvising and recognition of anomalies (Militello & Hutton 1998). A series of questions or probes that address 
each of the knowledge categories are used to obtain information about whether the particular skill is present in a 
task. In addition, strategies for its use and specific examples of its use are uncovered. The example is probed for the 
cues and strategies used in determining and dealing with the situation and potential errors a novice might make in 
the same position. A tape recording of each knowledge audit interview was reviewed and the data organized into a 
knowledge audit table. The table includes examples of where a particular skill (knowledge category) was used, the 
cues and strategies used in dealing with the situation and why the situation might be difficult for novices. 

Step 3. The Simulation Interview: In the simulation interview, information about the experts' cognitive processes 
within the context of a task is gathered. This is accomplished by the presentation of a challenging scenario to the 
expert. In our study scenario was "Full text search using a search engine" The biggest challenge with any type of 
on-line searching is choosing the right search terms or keywords. Because web search engine databases make no 
distinction between the various types of information available about a certain topic, doing a search about, say, a 
famous author, will probably get a hotchpotch of web sites, many of them personal home pages created by fans of 
the said author. 

Step 4. Sorting Out Key (Cognitively Demanding) Tasks: The final stage of the ACTA process is the construction 
of a cognitive demands table to sort through and analyze the data collected in the previous three stages. This step 
allows the less relevant information to be filtered, thus, focusing the analysis only on specific goals. 

The ACTA analysis in our study revealed the difficult cognitive elements in Internet tasks, the kinds of problems an 
expert expects when performing these tasks, strategies for solving problems and why the tasks might present 
problems to novices. These types of information are important components of the conceptual models to be 
developed. Presenting the information to novices in conceptual models is a fundamental aspect of constructivist 
learning (Brandt 1997). The data collected during the ACTA analysis are used to develop the conceptual models that 
would be implemented in the constructivist learning environments for novices to learn about Internet searching. 



Representation of the Conceptual Models 



We believe that analogies are powerful representations to help learners understand complex tasks such as searching 
the Internet. Analogies can give the novices a visualization that is very easy to relate to and they force the novice to 
compare and contrast to understand the analogies. The analogies chosen should be related to existing mental models 
held by learners. Often the analogy alone is not enough to accurately convey the conceptual model. Learners must 
be encouraged to look at the similarities and dissimilarities between the analogy and the topic. This can be achieved 
by supplementing the analogy where necessary with appropriate technical information about the topic and with 
practice examples that lead the learner to compare and contrast. Analogies can be tailored to help the learner acquire 
problem-solving skills. 

In order to understand the (unfortunately common) problems associated with loading web pages, some of the 
underlying mechanism of the transfer of data must be incorporated into the conceptual model. In this case, major 
concepts include the client/server model, the need to make a request of the server for a web page and the subsequent 
reply from the server (either an error message or the requested page). Possible problems when requesting a web page 
can include: the original request being lost, an extremely slow response from the server, a time-out of the original 
request, the reply from the server getting lost or timing out, etc. Therefore, the conceptual model must address the 
fact that problems can occur with firstly the request and then the reply. 
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An example of an analogy we developed to convey this was that of ordering a pizza over the telephone from a pizza 
restaurant. In this analogy, the person ordering is the client, the restaurant is the server and the pizza would represent 
the web page or file that was to be downloaded. Initially the person must make a request or order the pizza. To do 
this they need to know the telephone number of the restaurant. On ringing the number, one possible problem is that 
it will be engaged. This is analogous to a server not replying to the initial request for a web page and illustrates that 
the request for a web page has to reach the server to be satisfied, but may possibly 'fall at the first hurdle'. One 
reason for a request not reaching a server is that the server is extremely busy dealing with other requests. Again, the 
engaged line can convey this possibility. This analogy can be particularly useful in helping users interpret error 
messages that they receive whilst browsing. Rather than implicitly accept that an HTTP 404 error means that the 
page requested does not exist, or that a server is down or inaccessible, the user can interpret it as a busy line to be 
tried again. Redialling the number of the restaurant or re-clicking the hyperlink a few times often cures the problem. 

The pizza restaurant analogy can also help to explain other possible problems in the delivery of a web page to the 
client computer. If the pizza restaurant is very busy with lots of orders (requests for web pages) to deal with, it is 
likely that the order take longer to be fulfilled and despatched. With the Internet, as with pizza, patience is the key. 
The speed of delivery is not only influenced by how busy the restaurant (server) is, but also by the amount of traffic 
on the roads between the restaurant and the client's house (computer). If the traffic is very busy on the roads (or the 
networks that constitute the Internet), it is likely that it will take longer for the delivery to arrive. Furthermore, there 
is always the chance that the delivery person will get lost, as can happen with the reply from the server. 

According to Jonassen (1997), a good conceptual model contains a visual representation of all of the essential parts, 
states, or actions encountered in the problem and the relationships between them at a level of detail and familiarity 
that are appropriate for the learners. 



Web-Based Courseware 



A web courseware application based on the conceptual models derived for both Internet Explorer and Netscape was 
built by us for novices to use. This provided a learning environment in which novices could learn about searching on 
the Internet through interaction with the courseware. The exercises were also designed along constructivist 
principles to reinforce the conceptual models. Topics for novices included hyperlinks, URLs, HTML, the history 
mechanism, cut & paste, searching and web site structure and organisation (including home pages). In order to 
address the need for technical knowledge, the conceptual model presented of the task of following a hyperlink, for 
example, is described not only by an analogy, but also by concepts that underline the loading of a web page, such as 
client/server and request/reply protocols. Procedural knowledge is also supplied where this constitutes 'tricks of the 
trade' such as the use of the right mouse button as a shortcut to access context-sensible menus. 



Evaluation 



The courseware was given to a group of novices to evaluate. Twenty-six subjects took part. To evaluate the 
courseware module, they were each given the same searching exercise to undertake, working alone, using the 
courseware module. Following the learning exercise each was given a questionnaire to answer, following the 
exercise. The questionnaire used multiple choice questions, as a 5 point ranking, to elicit the novices opinions on 
ease of navigation through the module, how easy they found the module to follow, its balance, comprehensiveness 
and interest, together with Yes / No questions on whether using it had boosted their understanding, their knowledge 
and their confidence in Internet searching. Comments about their own skill levels and about aspects not addressed 
by other questions were also elicited. 

Only 4% of subjects had difficulty in navigating through the module, with 96% of subjects giving the top two 
rankings for these questions. 73% judged the module easy to follow giving the top two rankings, while 92% gave the 
top three rankings. 
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The questions addressing the effects of the module on the learning outcomes for the subjects found that 65% 
considered that it had increased their understanding, 60% that it had increased their knowledge but only 54% that it 
had increased their confidence in searching on the Internet. The first two outcomes are lower than we had expected, 
but may be due to a greater initial level of capability than we had measured. The figure for increase in confidence is 
more disappointing and would be difficult to explain in terms of already high confidence of subjects in their ability. 
Our experiences with the group do not support this. It may be that we administered the evaluation questionnaire too 
soon after the learning experience, before the subjects had the chance to develop their new skills through practice. 

Generally, we may claim that the majority of the novices found the learning environment useful in helping them to 
understand the basic principles of searching on the Internet. These novices also confirmed that the conceptual 
models were useful and helped them to have a better understanding of the search process involved and how to locate 
necessary information. 



Conclusion 



Successful Internet use requires problem-solving skills and technical concepts that are only acquired through 
experience. We have demonstrated in our case study that using appropriate conceptual models that are connected to 
users' mental models can help novices learn searching on the Internet. It is our belief that the idea of the use of 
carefully derived conceptual models stimulates learners to think about the similarities, or differences, which 
conceptual models describe. Using conceptual models that are connected to their mental models helps learners 
predict or solve problems. 

Further evaluations are planned to provide more convincing evidence of the effectiveness of the use of the 
conceptual models used in the courseware. Our experiment illustrates that the development of a conceptual model 
is not a trivial matter and much further research is required. However, it has demonstrated that a constructivist 
approach to courseware development using conceptual models in learning can help novices to acquire an 
understanding of the Internet search mechanism. 
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